feat: use a more robust queue system, with retry #54
Conversation
Will retry all HTTP request not-returning a 2xx status code
| @@ -8,3 +8,4 @@ SERVICE_PUSHER_BEAMS_INSTANCE_ID= | |||
| SERVICE_PUSHER_BEAMS_SECRET_KEY= | |||
| HUB_URL= | |||
| SNAPSHOT_URI= | |||
| REDIS_URL= | |||
There was a problem hiding this comment.
Hi @wa0x6e It won't work without Redis? Redis is usually expensive, retry can't be in memory or on mysql DB?
There was a problem hiding this comment.
Most of the light queue libraries uses Redis as backend, it's built for that.
Redis is not used to store the retry, but is used as storage solution for the whole queue system.
As for the redis instance, only a small instance will be required, as data will be cleared as queue is cleared, only data for failed/waiting for retry jobs will be stored.
Another repo (envelop) is also using the queue system, we could share the same redis instance for all repo using a queue system
There was a problem hiding this comment.
For that we need @bonustrack 's approval 😢 because he denied Redis a few times as far as I remember
@bonustrack what you think? we could use Redis here?
There was a problem hiding this comment.
I see, I've seen REDIS_URL in a few places, so I just assumed that it was common.
Issues
Handle all queue-like events with a true dedicated queue system, instead of using recursive
run()function withsleep.The queue system will allow more robust queue processing, such as dedicated process for each job and retry feature (fix #53), as well as easier testing.
Changes
Refactoring has been kept to the minimum, as workers are just used to trigger the function in the existing files, instead of moving the logic inside those workers file.
Todo
Notes
This PR is based on #60 and #61, and may contains unrelated changes until these are merged